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Abstract. To determine the stability and instability of a given steady galaxy 
configuration is one of the fundamental problems in the Vlasov theory for 
galaxy dynamics. In this article, we study the stability of isotropic spherical 
symmetric galaxy models fo(E), for which the distribution function /o depends 
on the particle energy E only. In the first part of the article, we derive the first 
sufficient criterion for linear instability of fo{E) : fo{E) is linearly unstable if 
the second-order operator 



has a negative direction, where V is the projection onto the function space 
{g(E, L)}, L being the angular momentum [see the explicit formulae l|27|l and 
H26H . In the second part of the article, we prove that for the important King 
model, the corresponding Aq is positive definite. Such a positivity leads to the 
nonlinear stability of the King model under all spherically symmetric pertur- 
bations. 



A galaxy is an ensemble of billions of stars, which interact by the gravitational 
field which they create collectively. For galaxies, the collisional relaxation time is 
much longer than the age of the universe ([8]). The collisions can therefore be 
ignored and the galactic dynamics is well described by the Vlasov - Poisson system 
(collisionless Boltzmann equation) 



where (x, v) € M 3 x R 3 , f(t,x,v) is the distribution function and Uf(t,x) is its 
gravitational potential. The Vlasov-Poisson system can also be used to describe the 
dynamics of globular clusters over their period of orbital revolutions One of 

the central questions in such galactic problems, which has attracted considerable 
attention in the astrophysics literature, of [7j, [E], [11], [31] and the references there, 
is to determine dynamical stability of steady galaxy models. Stability study can be 
used to test a proposed configuration as a model for a real stellar system. On the 
other hand, instabilities of steady galaxy models can be used to explain some of the 
striking irregularities of galaxies, such as spiral arms as arising from the instability 
of an initially featureless galaxy disk ([7]), ([52]). 

In this article, we consider stability of spherical galaxies, which are the simplest 
elliptical galaxy models. Though most elliptical galaxies are known to be non- 
spherical, the study of instability and dynamical evolution of spherical galaxies 
could be useful to understand more complicated and practical galaxy models . By 
Jeans's Theorem, a steady spherical galaxy is of the form 




1. Introduction 



(1) d t f + v V x f- X7 X U- V„/ = 0, 




fo(x,v) = f (E,L 2 ) 
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where the particle energy and total momentum are 

E=^\v\ 2 + U (x), L 2 = \xxv\ 2 , 

and Uq(x) = Uo(\x\) satisfies the self-consistent Poisson equation. The isotropic 
models take the form 

f (x,v) = f (E). 

The cases when Jq(E) < has been widely studied and these models are known 
to be linearly stable to both radial ([9]) and non-radial perturbations ([2]). The 
well-known Casimir-Energy functional (as a Liapunov functional) 

(2) H(f) =J J Q(f) + \J J \v\ 2 f - ± J IV^I 2 , 

is constant along the time evolution. If j' Q {E) < 0, we can choose the Casimir 
function Qq such that 

Q'ME)) = -E 

for all E. By a Taylor expansion of Tt(f) — Tt(fo), it follows that formally the first 
variation at /o is zero, that is, (fo(Ej) = (on the support of f (E)), and the 
second order variation of TL at / is 

where Q"(fo) = zzjrrm > 9 = f ~ fo an d AJ7 g = J gdv. In the 1960s, Antonov (pQ, 
[2]) proved that 

(4) < [Dh] = U ^dxdo -±J | V^| 2 dx 

is positive definite for a large class of monotone models. Here 

D = v ■ V x - VxE/q • V„, 

h(x,v) is odd in v and —Aip = J Dhdv. He showed that such a positivity is 
equivalent to the linear stability of fo(E). In [}J], Doremus, Baumann and Feix 
proved the radial stability of any monotone spherical models. Their proof was 
further clarified and simplified in [TU], [37], (55], and more recently in [33], [2T] . 
In particular, this implies that any monotone isotropic models are at least linearly 
stable. 

Unfortunately, despite its importance and a lot of research (e.g., [20], [3], [6], 
[13j). to our knowledge, no rigorous and explicit instability criterion of non-monotone 
models has been derived. When fb(E) changes sign, functional Ttj o is indefinite 
and it gives no stability information, although it seems to suggest that these mod- 
els are not energy minimizers under symplectic perturbations. In this paper, we 
first obtain the following instability criterion for general spherical galaxies. For 
any function g with compact support within the support of fo(E), we define the 
\fh(E) \ —weighted L 2 (R 3 x R 3 ) space Lj^,. with the norm |H||y/| as 

(5) W h Wh = [ I \fo(E)\h 2 dxdv. 
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Theorem 1.1. Assume that fo{E) has a compact support in x and v, and /q is 
bounded. For <ft £ H\ define the quadratic form 

(6) {A 4>,<f>) = J ' |V0| 2 dx + 4tt J f ti(E)(4,-V4>) 2 dxdv, 

where V is the projector of L 2 , , to 

\f'o\ 

kerD = {g(E,L 2 )}, 

and more explicitly V(f) is given by 118\) for radial functions and i26\) for general 
functions. If there exists (f>o € H 1 such that 

(7) (A o 0o,^o)<O, 

then there exists Ao > and <fr € H 2 , f (x,v) given by \14\) , such that e A °'[/, <j>] is 
a growing mode to the Vlasov-Poisson system (Qp linearized around [fo(E),Uf Q ] . 

A similar instability criterion can be obtained for symmetry preserving pertur- 
bations of anisotropic spherical models fo (E,L 2 ), see Remark[2j We note that the 
term V(f> in the instability criterion is highly non-local and this reflects the collective 
nature of stellar instability. The proof of Theorem ll.il is by extending an approach 
developed in [5S] for ID Vlasov-Poisson, which has recently been generalized to 
Vlasov-Maxwell systems ([2B], [IB])- There are two elements in this approach. One 
is to formulate a family of dispersion operators A\ for the potential, depending 
on a positive parameter A. The existence of a purely growing mode is reduced to 
find a parameter Ao such that the A\ has a kernel. The key observation is that 
these dispersion operators are self-adjoint due to the reversibility of the particle 
trajectories. Then a continuation argument is applied to find the parameter Ao 
corresponding to a growing mode, by comparing the spectra of A\ for very small 
and large values of A. There are two new complications in the stellar case. First, 
the essential spectrum of A\ is [0, +oo) and thus we need to make sure that the 
continuation does not end in the essential spectrum. This is achieved by using some 
compactness property due to the compact support of the stellar model. Secondly, 
it is more tricky to find the limit of A\ when A tends to zero. For that, we need an 
crgodic lemma fLemma l2.4[) and use the integrable nature of the particle dynamics 
in a central field to derive an expression for the projection V<j> appeared in the limit. 

In the second part of the article, we further study the nonlinear (dynamical) 
stability of the normalized King model: 



(8) fo = [e E °- E 1] + 

motivated by the study of the operator Aq. The famous King model describes 
isothermal galaxies and the core of most globular clusters [21] ■ Such a model 
provides a canonical form for many galaxy models widely used in astronomy. Even 
though /q < for the King model, it is important to realize that, because of the 
Hamiltonian nature of the Vlasov-Poisson system |T]) , linear stability fails to imply 
nonlinear stability (even in the finite dimensional case) . The Liapunov functional is 
usually required to prove nonlinear stability. In the Casimir-energy functional |2]), it 

(2) 

is natural to expect that the positivity of such a quadratic form T~Cf [g] should imply 
stability for fo(E). However, there are at least two serious mathematical difficulties. 

(2) 

First of all, it is very challenging to use the positivity of [g] to control higher 



4 



YAN GUO AND ZHIWU LIN 



order remainder in H(f) —H{fo) to conclude stability [38] . For example, one of the 
remainder terms is f 3 whose L 2 norm is difficult to be bounded by a power of the 
stability norm. The non-smooth nature of fo(E) also causes trouble here. Second of 

(2) 

all, even if one can succeed in controlling the nonlinearity, the positivity of H j o [g] 
is only valid for certain perturbation of the form g = Dh [25] . It is not clear at all 
if any arbitrary, general perturbation can be reduced to the form Dh. To overcome 
these two difficulties, a direct variational approach was initiated by Wolansky [55] . 
then further developed systematically by Guo and Rein in [14] , [15] , [17] , [18] , [19] . 
Their method avoids entirely the delicate analysis of the second order variation 
7Yj o in ([3]), which has led to first rigorous nonlinear stability proof for a large class 
of fo(E). The high point of such a program is the nonlinear stability proof for every 
polytrope [18] fo(E) — (Eq — Eyl_. Their basic idea is to construct galaxy models 
by solving a variational problem of minimizing the energy under some constraints 
of Casimir invariants. A concentration-compactness argument is used to show the 
convergence of the minimizing sequence. All the models constructed in this way 
are automatically stable. 

Unfortunately, despite its success, the King model can not be studied by such a 
variational approach. The Casimir function for a normalized King model is 

(9) Q (/) = (l + /)ln(l + /)-l-/, 

which has very slow growth for / — > oo. As a result, the direct variational method 
fails. Recently, Guo and Rein [21] proved nonlinear radial stability among a class 
of measure-preserving perturbations 
(10) 



f(t,r,v r ,L)>0: JqU,E) = J Q(f , L), for Q E and Q(0, L) = 0. 



The basic idea is to observe that for perturbations in the class <S/ , one can write 
g = f-f asDh= {h, E}. Therefore, TifJ [g] = TifJ [Dh], for which the positivity 
was proved in (S3] for radial perturbations. To avoid the difficulty of controlling the 
remainder term by T~if [g] , an indirect contradiction argument was used in [21j . 

As our second main result of this article, we establish nonlinear stability of King's 
model for general perturbations with spherical symmetry: 

Theorem 1.2. The King's model fo = [e Eo ~ E — 1]+ is nonlinearly stable under 
spherically symmetric perturbations in the following sense: given any e > there 
exists ei > such that for any compact supported initial data /(0) £ C\ with 
spherical symmetry, if d (/ (0) , fo) < S\ then 

sup d(f(t) ,/ ) < e, 

0<t<oo 

where the distance functional d(f,fo) is defined by \35\) . 

For the proof, we extended the approach in [27] for the 1^-D Vlasov- Maxwell 
model. To prove nonlinear stability, we study the Taylor expansion of TL{f) — Ti(fo). 
Two difficulties as mentioned before are: to prove the positivity of the quadratic 
form and to control the remainder. We use two ideas introduced in |27j . The first 
idea is to use any finite number of Casimir functional Qi (/, L 2 ) as constraints. 
The difference from [3T] is that we do not impose Qi (/, L 2 ) = Qi (/o, £ 2 ) in the 
perturbation class, but expand the invariance equation Qi (/ (t) , L 2 )~Qi (fo, L 2 ) = 
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Qi (/ (0) , L 2 ) — Qi (/o, L 2 ) to the first order. In this way, we get a constraint for 
g = f — fo in the form that the coefficient of its projection to d\Qi (fo, L 2 ) is small. 
Putting these constraints together, we deduce that a finite dimensional projection of 
g to the space spanned by {d±Qi (/o, £ 2 ) } is small. To control the remainder term, 
we use a duality argument. Noting that it is much easier to control the potential 4>, 
we use a Legendre transformation to reduce the nonlinear term in g to a new one 
in <p only. The key observation is that the constraints on g in the projection form 
are nicely suited to the Legendre transformation and yields a non-local nonlinear 
term in <j) only with the projections kept. By performing a Taylor expansion of this 
non-local nonlinear term in tfi, the quadratic form becomes a truncated version of 
(Ao4>, <f) defined by ([6]), whose positivity can be shown to be equivalent to that of 
Antonov functional. The the remainder term now is only in terms of <fi and can 
be easily controlled by the quadratic form. The new complication in the stellar 
case is that the steady distribution fo (E) is non-smooth and compactly supported. 
Therefore, we split the perturbation g into inner and outer parts, according to the 
support of fo. For the inner part, we use the above constrainted duality argument 
and the outer part is estimated separately. 



2. An Instability Criterion 
We consider a steady distribution 

f (x,v) = f (E) 

has a bounded support in x and v and /q is bounded, where the particle energy 
E = \\v\ 2 + Uo(x). The steady gravitational potential U~o{x) satisfies a nonlinear 
Poisson equation 

AC/ = ^ J fodv. 
The linearized Vlasov-Poisson system is 

(11) d t f + v V x f - V x Uo ■ V v f = V x <t> ■ V„/o, A^ = 4tt J f(t, x, v)dv. 

A growing mode solution (e xt f(x, v), e xt (f)(x)) to |T]) with A > satisfies 

(12) A/ + v ■ V x f - V X U Q ■ V v f = • V«0. 
We define [X(s; x, v), V(s; x, v)] as the trajectory of 



(13) 



such that X(0;x,v) — x, and V(0;x,v) = v. Notice that the particle energy E is 
constant along the trajectory. Integrating along such a trajectory for ~oo < s < 0, 
we have 

,o 

(14) f(x,v)= e Xs f^E)V(s;x,v)-V x ^(X( S ;x,v))d S 

J — OO 

= f' {E)<t>{x) - f Q {E) I Xe Xs cf>(X( S ; x, v))ds. 
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Plugging it back into the Poisson equation, we obtain an equation for <j> 

-A(j> + [4tt J f {E)dv]<j) - 4tt J f' {E) J Xe Xs (f>(X(s; x, v))dsdv = 0. 

We therefore define the operator A\ as 

A\<j) = —A(f> + [4tt J fo(E)dv]<j> - 4tt J f (E) j Xe Xs <j)(X(s;x,v))dsdv. 

Lemma 2.1. Assume that fo(E) has a bounded support in x and v and f' is 
bounded. For any A > 0, the operator A\ : H 2 — > L 2 is self-adjoint with the 
essential spectrum [0, +oo) . 

Proof. We denote 

K x <jy = -4tt[ J f (E)dv}<j> + 4tt J f (E) j \e Xs <j>(X (s; x, v))dsdv. 

Recall that fo{x,v) = fo{E) has a compact support C S C M 3 , x K 3 . We may 
assume S = S x x S v , both balls in R 3 . Let y = X i\ x \) be a smooth cut-off function 
for the spatial support of fo in the physical space S x ; that is, x = 1 on the spatial 
support of fo and has compact support inside S x . Let M x be the operator of 
multiplication by x- Then K x = K\M X = M X K X = M X K\M X . Indeed, 

fo ( x > v ) = fo (X(s;x,v),V(s;x,v)) 
because of the invariance of E under the flow. So 

(15) (K x <j>) (x) = -4tt[ J f' Q {E)dv]<t> + 4ttJ f (E) J Xe Xs <jy(X(s; x, v))dsdv 

= -4tt[ J UE)dv]4> + 4nJ J Xe Xs (ti(E)<j>) (X(s; x, v))dsdv 
= (M x K x M x 0)(x). 



First wc claim that 



\\Kx\\ L ^ L2 < Stt 



/ \fo(E)\ 



dv 



Indeed, the L 2 norm for the first term in K\ is easily bounded by 47r | J f^(E)dv\ 

For the second term, we have for any ^eL 2 , 

(16) 



[ [ 4n\e Xs f' Q {E)(t>{X{s-x,v))dsdv4>{x)dx\ 

-oo J J 

<4irJ Xe Xs (^J J \ti(E)\4?(X(s;x,v))dvdx^ ' (^J J \f (E)\^ 2 (x)dvdx^j ' ds 
= 4nJ° Xe Xs (^j J \f( ) (E)\(b 2 (x)dvdxy (^J J \f (E)\^ 2 (x)dvdxY ds 

1 1 

= ^ (/ j \f' Q {E)\<t> 2 {x)dvdxy (/ I \f' {E)\i> 2 (x)dvdx 



< 4?r 



/ l/o(^)l 



dv 
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Moreover, we have that K x is symmetric Indeed, for fixed s, by making a change of 
variable (y, w) — > (X(s; x, v), V(s; x, v)), so that (z, v) = (X(—s; y, w), V(— s; y, w)), 
we deduce that 

r° 

info{E) / Xe Xs (/)(X(s;x,v))dsdvip(x)dx 

J — OO 

Xe Xs J J 47rti(E)0(y)iP(X(- S ;y,w))dydwd S 
inf^E) / Xe Xs ^(X(-s;y,-w))4>(y)dydwds 

r° 

4tt/o(£;) / \e Xs ip(X(s;x,v))(/)(x)dvdxds. 

J — oo 

Here we have used the fact [X(s;y,w),V(s;y,w)} = [X(— s; y, — w), — V(s; y, — w)] 
in the last line. Hence 

{K X M) = {4>,K > $). 

Since K\ = K\M X and M x is compact from H 2 into L 2 space with support in S x , 
so K\ is relatively compact with respect to —A. Thus by Kato-Relich and Weyl's 
Theorems, A x : H 2 — > L 2 is self-adjoint and ct css (Aa) = cr css (— A). □ 

Lemma 2.2. Assume that fb(E) has a bounded support in x and v and /q is 
bounded. Let 

k(X)= inf {4>,A X <P), 

then k(X) is a continuous function of X when A > 0. Moreover, there exists < 
A < oo such that for A > A 

(17) fc(A) > 0. 

Proof. Fix A > 0, (j) e D(A X ), and \\4>\\ 2 = 1. Then 

fc(Ao) < (<Ma £) 

< {<t>,A x <l>) + \{<t>,A XQ <l)) -{<!>, A x 4>)\ 

< (4>,A X <I>) J\fo(E)\J [Xe Xs - X e x « s }<j)(X(s;x,v))<j)(x)dsdvdx 

< (<M*0) +4tt f [\fo(E)\[ [ [X\s\e Xs + e Xs }dX(t>(X(s;x,v))(t>(x)dsdvdx 

J J J — OO J Xq 

pX 

As , ^Asi 



< (<t>, A X (f>) + C / [A|s|e As +e As ]dAds 

J — OO J Xq 

< (cj>,A x <p) + C\hiX- In X \. 

We therefore deduce that by taking the infimum over all <j>, 

fc(A ) < fc(A) + C|lnA-lnA |. 

Same argument also yields fc(A) < fc(Ao) + C\ InA — lnAo|.Thus |fc(Ao) — fc(A)| < 
C| In A — In Ao| and fc(A) is continuous for A > 0. 

To prove (fT7|). by lfl4|) . we recall from Sobolev's inequality in R 3 
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\(K\4>,ip)\ = J J ±nf' Q {E)e Xs V4>{X{s;x,v))V{s)dsdw4>{x)dx 

< f_J Xa (/ / |^| 2 |/o(^)l^) 1/2 - 

x[[ [ {V^X (s))\ 2 \f'(E)\\V ^fdxdv^ds 



•o / r r \ 1/2 



.As 



<y||^||6l|V^|| 2 <y||V^|| 2 ||V0|| 2 , 
since /o has compact support. Therefore, 

C 



\^\ 2 \f' {E)\dvdx) / / v 2 \V^x)\ 2 \f (E)\dxdv]^ 2 ds 



(A x ^,cb) = ||V0|| 2 - (KxM) > (1- y)||V0|| 2 > 

for A large. □ 

We now compute lim^o+ A\ . We first consider the case when the test function 
<fi is spherically symmetric. 

Lemma 2.3. For spherically symmetric function 4>{x) = , we have 

(18) 

lim (A x cj),(j)) = (A <f>,(f>) = / |V</>| 2 da; + 47r / / f' Q {E)dvcf) 2 dx 



/ rr 2 (E,L) gdr \ 

, f E r°° , {JrtiE.L) J 2 (E-U Q -L 2 /2r 2 ) ) 

-32tt 3 / fo(E)- , p n V -^-dLdE 

■ rr n f-r*(E,L) dr 

JmmUo JO I ,„ , > — . 

Jr i( E , L ) v / 2(E-U -L 2 /2r 2 ) 

r r rr 2 (E,L) drdEdL 

- 1 ^ + 32 * s / ^ L. L , <* - ^vwmjwy 

Proof. Given the steady state fo(E), Uo(\x\) and any radial function </>(|x|) . To 
find the limit of 



(19) (A x <l>,4>) = j |V0| 2 dx + 47r / / f[,{E)dv4, 2 dx 

-4tt J J fo(E) (^J Xe Xs (f>(X(s;x,v))ds j 4>{x)dxdv, 
we study the following 

,0 

(20) lim / Xe Xs (/)(X(s;x,v))ds. 

A ^°+ J-oo 

Note that we only need to study ([20|) for points (x, v) with E = i|v| 2 + £/o| (x|) < Eq 
and L — \x x v\ > 0, because in the third integral of (TTO|) f' Q {E) has support in 
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{E < Eq} and the set {L = 0} has a zero measure. We recall the linearized Vlasov- 
Poisson system in the r,v r , L coordinates takes the form 

d t f + v r drf + \-j - d r Uo) d v J = d r Ufd Vr f , 



d rr Uf + ^d r Uf = 4tT / fdl 



For the corresponding linearized system, for points (x, v) with E < Eq and L > 0, 
the trajectory of (X(s;x, v),V(s;x, v)) in the coordinate (r,E,L) is a periodic 
motion described by the ODE (see [8]) 

dr(s) 

—— = V r (s), 

as 

with the period 



T(E,L) = 2 f 

J r 



l{E ,L) ^2(E - U - L 2 /2r 2 )' 

where < r\(E,L) < r 2 (E,L) < +oo are zeros of E — Uq — L 2 /2r 2 .So by Lin's 
lemma in [|25j]. 

r° i r T 

lim/ Xe Xs (j)(X(s;x,v))ds = — <f)(X(s;x,v))ds. 
Since <fi(X(s; x, v) — <fi(r(s)), a change of variable from s — ► r(s) leads to 
%(X(.;x,t,))* = 2 ^ )dr 



n y/2{E - U - L 2 /2r 2 ) 
For any function g(r, E, L), we define its trajectory average as 

f-r 2 (EX) g(r,E,L)dr 



g(E,L) 



Jri 



( E > L ) y/2(E-U -L 2 /2r 2 ) 



f 



r 2 (E,L) dr 



i E , L ) y / 2(E~U -L 2 /2r 2 ) 



Then 



lim / \e Xs <P{X{s-x 1 v))ds = 2 f * - IT (E,L) = <j,{E,L) 

^v+J-oo Jri ^2(E-U Q -L 2 /2r 2 )' V ' ' ^ V 7 
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and the integrand in third term of (fT9|) converges pointwise to /q(P)00. Thus by 
the dominated convergence theorem, we have 



lim + (A A <£, 4>) = J \Vcj)\ 2 dx + ATT J J f' {E)<j) 2 'dxdv - 4tt J J fo{E)4><f> dxdv 
\X7cj)\ 2 dx + 4:ir J J f' (E)<t> 2 dxdv 



■ f E f°° s f ME ' L) - , \ i \ drdEdL 
32tt 3 / / fo(E) / q>(E,L)^(r)- 



IminUoJO Jn(E,L) ' V%(E — Uq — L/2r 2 ) 

= I \\74>\ 2 dx + 4n I I fo(E)(f> 2 dxdv 



E 



rr 2 (E,L) 0rfr 



PT-2 

., , , Jri(E,L) v /2(E-U -L/2r2) . 

32 - 3 / / ' rM E,L) dr h dEdL 

J min Uq J I , _ r > — = 

J ri(E,L) ^2(E-U -L/2r 2 ) 

iv0i 2 + 32^ 3 y y 



/n(B,L) ^2(£; - C/ - i/2r 2 ) 

This finishes the proof of the lemma. □ 

To compute lirnx-^o+CAx^, 0) for more general test function </>, we use the fol- 
lowing ergodic lemma which is a direct generalization of the result in [26j . 

Lemma 2.4. Consider the solution (P (s;p,q) ,Q (s;p,q)) to be the solution of a 
Hamiltonian system 

P = d q H(P,Q) 
Q = -d p H (P, Q) 

with (P(0),Q(0)) = (p,q) G R™ x R". Denote 

r° 

Q x m = / Xe Xs m(P(s) ,Q(s))ds. 

Then for any m(p,q) G L 2 (R" x R"), we have Q x m — > Pm strongly in L 2 (R" x R"). 
Pere P is i/ie projection operator of L 2 (R™ x R") to </ie kernel of the transport 
operator D — d q Hd p — d p Hd q and Vm is the phase space average of m in the set 
traced by the trajectory. 

Proof. Denote U (s) : L 2 (R" x R") -> L 2 (R n x R n ) to be the unitary semigroup 
U (s)m = m(P (s) , Q (s)). By Stone Theorem ([10]), U (s) is generated by iR = D, 
where R = —iD is self-adjoint and 

/ + OC 
e las dM a 
-oo 

where {M a ; a G R 1 } is spectral measure of R. So 

f Xe Xs m(P(s),Q(s))ds = [ Xe Xs [ e ias dM a mds= [ A dM a m. 
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On the other hand, the projection is V = A/{o} = Jjg.£dM a where £(a) = for 
a ^ and £(0) = 1. Therefore 



2 ' A 



Xe Xs m(P(s), Q(s))ds - Vm 



A + ia 



d\\ M a m\ 



L 2 



L 2 

by orthogonality of the spectral projections. By the dominated convergence theo- 
rem this expression tends to as A — > 0+, as we wished to prove. The explaination 
of Vm as the phase space average of m is in our remark below. □ 

Remark 1. Since f_ \e Xs ds — 1, the function 

,o 

(21) (Q X m){x,v) = Xe Xs m(P(s),Q{s))ds 

J — oo 

is a weighted time average of the observable m along the particle trajectory. By the 
same proof of Lemma \2.4\ we have 

1 f T 

(22) lim - / m(P(s),Q{s))ds = Vm. 
T-,00 T J 

But from the standard ergodic theory (^) of Hamiltonian systems, the limit of the 
above time average in V22\) equals the phase space average of m in the set traced 
by the trajectory. Thus Vm has the meaning of the phase space average of m and 
Lemma \2.4\ states that the limit of the weighted time average \21)) yields the same 
phase space average. In particular, if the particle motion is ergodic in the invariant 
set Sj determined by the invariants E\,--- , and if daj denotes the induced 
measure of R™ x R™ on Si, then 

(23) Vm = [ m (p, q) doi (p, q) . 

°7 (Si) J Si 

For integral systems, using action angle variables (Jx, • • • , </„; tpi, • • • , tp n ) we have 
(24) 

{Vm) (Jx, ■ ■ ■ , J n ) = (2vr) _ ™ / ■••/ m (J%, ■ ■ ■ , J n , <pi, ■ ■ ■ , ip n ) dipx, • • • d(p n 



for the generic case with independent frequencies (see [J],). 

Recall the weighted L 2 space Lf , in Then U (s) : Lf , — » Lf , defined by 

U (s)m — m (X(s; x, v), V(s; x, v)) is an unitary group, where (X(s; x, v), V(s; x, v)) 
is the particle trajectory (fT3|) . The generator of U (s) is D — v ■ d x — V x Uq ■ V„ and 
R = —iD is self-adjoint by Stone Theorem. By the same proof, Lemma \2. 41 is still 
valid in L 2 ^y In particular, for any <p(x) £ L? (R 3 ) we have 

r° 

(25) / Xe Xs <j>(X(s;x,v))ds^V<t) 

J — oo 

in Lf ,, where V is the projector of Lf , to kerD. 

I Jo I |/o| 

Now we derive an explicit formula for the above limit V<fr. Note that as in the 
proof of lemma [2~3l we only need to derive the formula of V<f> for points (x, v) with 
E < E and L > 0. Since Uq (x) = Uq (r), the particle motion (Tf3| in such a center 
field is integrable and has been well studied (see e.g. [8], [4]). For particles with 
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energy E < Eq < 0, L > and momentum L = x x v, the particle orbit is a rosette 
in the annulus 

A e ,l = {ri(E, L)<r< r 2 (E, L)} = {E - U - L 2 /2r 2 > 0} , 

lying on the orbital plane perpendicular to L. So we can consider the particle 
motion to be planar. For such case, the action-angle variables are as follows (see 
e.g. [3U]): the actions variables are 

Jr = T^Ty Je = L ' 

where 

r«(B,£) dr 



T(E,L) = 2 f 

Jri 



ri (E,L) sj2{E -U - L 2 /2r 2 ) 
is the radial period, the angle variable ip r is determined by 

27r dr 
^ = T (E, L) ^2{E - U - L 2 /2r 2 / 
and ipg = 9 — A9 where 

d(A6) = - Lr "- n ° dr 
^2{E -U - £ 2 /2r 2 ) 

and 

1 pr 2 (EX) j 

n g (E,L) = — -/ dr 

T (E, L) J ri [E,L) r*y/2(E-U Q -L*/2r*) 

is the average angular velocity. For any function <fi (%) € H 2 (R- 3 ) , we denote 
4>£ (r, 9) to be the restriction of (f> in the orbital plane perpendicular to L. Then by 
(I24[) . for the generic case when the radial and angular frequencies are independent, 
we have 

(26) {V4>) (e,l) = ^)' 2 J J <f>L d( P<> d( Pr 

1 rr2(EX) ,2tt <j) Z {r,0)dJ9dr 



nT(E,L) J rx{E ,L) Jo y/2(E - U - L 2 /2r 2 ) ' 
In particular, for a spherically symmetric function <f> = <f>(r), we recover 

(27) m (E, L) = 2 t»™ Hr)dr 

K ' T(E,L)J ri{EtL) ^2(£-£/ -L 2 /2r 2 ) 

We thus conclude the following 

Lemma 2.5. Assume that fo(E) has a bounded support in x and v and /q is 
bounded. For any (j) € we have 

(28) 

lim (A x 4>, (t>) = (Ao(f>, 4>) 

A — ►0-|- 

= J |V<?!)| 2 da; + 47r / J fo(E)dv^) 2 dx - 4tt J J f' (E) (V(f>f dxdv 
\V(f>\ 2 dx + 4:Tr J J f' {E) {(j> - V<f)) 2 dxdv 
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where V is the projector of L? , to kerD and more explicitly V(f> is given by {£( 

I J o I 

The limiting operator A is 

(29) A cj) = -A0 + [4tt J f {E)dv]<j) - 4tt J f' Q {E)V (f>dv . 

Now we give the proof of the instability criterion. 
Proof of Theorem We define 

A* = sup A. 

fc(A)<0 

By Lemmas 12.11 and 12.51 we deduce that 

— oo < A» < A < oo. 
Therefore, by the continuity of fc(A), we have 

fc(A») = 0. 

Hence, there exists an increasing sequence of A„ < A„+i < A* so that A„ — * A*, 
k n = k(X n ) < 0, and 

k n — ► fc(A*) = 0. 

Therefore, k n are negative eigenvalues. By Lemma l2T2l we get a sequence 0„ G H 2 
such that 

(30) A\ n cj> n = k n (f>„ 

with k n < 0, k n — > and A„ — > Ao > 0, as n — > oo. Recall x the cutoff function 
of the support of fo(E) such that x = 1 for fo(E) > 0. We claim that x4>n 1S a 
nonzero function for any n. Suppose otherwise, x4>n = 0, then from the equation 
(|30[) we have (—A — k n ) <p n — which implies that (j> n — 0, a contradiction. Thus 
we can normalize ^ n by ||x0n|l2 = Taking inner product of (I30p with <f) n and 
integrating by parts, we have 

llV^lla ^ ~ 4?r / [ fo( E )<Pl d vdx+ [ [ inf^E) f \ n e x - s (p n {X{ S ;x,v))ds(p n {x)dx 



= -4tt / J fo(E) ( X <t> n y dvdx 

+ f f 4tt/£(£) [ A„e A " s ( X <l>n)(X(s;x,v))ds(x(t>n)(x)dx 



< 8tt 



f' {E)dv 



\xK 



2 



Here in the second equality above, we use the fact % = 1 on the support of 
f' {E) (f (E)) and that (x<t>n,) (X(s;x,v)) = 4> n (X{s\x,v)x due to the invariance 
of the support under the trajectory flow, as in (|15[) . In the last inequality, we use 
the same estimate as in (IT51). Thus, 



sup||0„|| L e < Csup||v<M 2 < c '' 

n n 

for some constant C independent of n. Then there exists <f> £ L 6 and € L 2 
such that 

0n - * 4> weakly in L 6 , and V0 n — > V0 weakly in L 2 . 
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This implies that x&n — * x4> strongly in L? . Therefore || x^ll 2 = 1 an( ^ thus 7^ 0. 
It is easy to show that cf> is a weak solution of A\ <p = or 





Ans, 



(31) - A0 = -[4tt / f' {E)dv\<j) + A-Kf a {E) I \ Q e x ° s (j>(X(s; x, v))dsdv = p 



We have that 



pdx = ~4irj J f' {E)4>{x)dxdv + J A e Aos y J 4iTfo{E)^(X(s;x,v))dxdvds 
= -4tt / / fo(E)<p (x) dxdv + [ X e x ° s [ [ 4nf( ) (E)(t)(x)dxdvds=0 



and by (|31|) p has compact support in S x , the x— support of fo(E). Therefore from 
the formula <j) (x) — J dy, we have 



*(*)=/ J^) d y = I J^-dy -f P -^dy = , 
J F — 2/| J F - 2/1 J Fl V / 



for x large, and thus <f> G L 2 . By elliptic regularity, <f> G iJ 2 . We define f(x,v) by 
(fT4|) . then / G L°° with the compact support in S. Now we show that e A °*[/, </>] is a 
weak solution to the linearized Vlasov-Poisson system. Since <f> satisfies the Poisson 
equation (131 [) , we only need to show that / satisfies the linearized Vlasov equation 
([T3]) weakly. For that, we take any g G C\ (R 3 x R 3 ) , and 



(Dg) fdxdv 



(Dg)(ti{E)<p(x))dxdv- 1 1 (Dg)ti(E) I \ e x ° s <j){X{ S] x,v))dsdxdv 

R 3 xR 3 







= 1 + 11. 



Since D is skew-adjoint, the first term is 



I = - [[ gD (ti(E)<p) dxdv = - ff f' Q {E)gD^dxdv. 

J./R 3 xR 3 JJR3 X R3 
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For the second term, 

11= - { A e Aos / / f' Q {E) Dg(x, v) 4> (X(s; x, v)) dxdvds 



— oo 




3 xR 3 

A e Aos / / f' {E) (Dg) (X(-s), V(-s)) <j> (x) dxdvds 

! xR 3 

/,',( E ) / A,„ A - (-j-g V(-s))\ ds cf> (x) dxdv 

j; i {E)\\ l)! i(.r.r)- I Xle^g (X(-s), V(-s)) ds) <f> (x) dxdv 
f! > (E)\u->[.r\ - ./;',(£') / Age A » > (X(s), V(s)) ds) g (x, v) dxdv 
A,,// {&{E)4>{x)-f {E) I \ Q e x ° s cf>(X(s),V(s))ds}gdxdv 

! xR 3 



— -Ao / / fgdxdv. 

././r 3 xR 3 

Thus we have 

ff (Dg)fdxdv= ff (A / - f Q {E)Dif>) gdxdv 

JJR 3 xR 3 JJR 3 xR 3 

which implies that / is a weak solution to the linearized Vlasov equation 

Ao/ + Df = f (E) v ■ V x <p. 

□ 

Remark 2. Consider an anisotropic spherical galaxy with f (x,v) = fo (E, £ 2 )- 
For a radial symmetric growing mode e At (<ft, f) with <j> = <f> (\x\) and f — f [\x\ , E, L 2 ^j 
The linearized Vlasov equation becomes 

\f + v-V x f-V x U -V v f 
= V x ■ V t ,/ = ■ + J^V, (|a; x v\ 2 ) 

which is of the same form as in the isotropic case h2U\). So by the same proof of The- 
orem [7771 we also get an instability criterion for radial perturbations of anisotropic 
galaxy, in terms of the quadratic form U8\) with fb(E) being replaced by ^jjr. 

3. Nonlinear Stability of the King's Model 

In the second half of the article, we investigate the nonlinear stability of the King 
model ([8]). We first establish: 

Lemma 3.1. Consider spherical models fo = fo (E) with f < 0. The operator 
Aq : H 2 — ► £ 2 

A 4> = -A<j> + [4tt / f' dv]cp -Air / fiP<t>dv 
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is positive, where H 2 and L 2 are spherically symmetric subspaces of H 2 and L 2 
and the projection V4> is defined by \2T\). Moreover, for <j> € H 2 



(32) (A)^0)>e(|V^ + |^) 

for some constant e > 0. 

Proof. Define ko = inf (Ao<f), cf>) / (cf>, <f>) .We want to show that fco > 0. First, by 

using the compact embedding of H 2 <—> L 2 it is easy to show that the minimum 

can be obtained and ko is the lowest eigenvalue. Let Ao<fio — ko<fio with </>q 6 H 2 

and 1 1 <^>o 1 1 2 = 1< The fact that ko > follows immediately from Theorem 11.11 and the 

nonexistence of radial modes ([9], [22]) for monotone spherical models. The proof 

of ko > is more delicate. For that, we relate the quadratic form (Aq4>, 4>) to the 

Antonov functional (TJ|. We define D — v ■ d x — V x Uo ■ V„ to be the generator of the 

unitary group U (s):L, — ► L,']",, defined by U(s)m — m (X(s; x, v), V(s; x, v)) . 
\fo\ \fo\ 

Here i s the spherically symmetric subspace of which is preserved under 

the flow mapping U (s). By the definition of Pcf>, we have 4>o — V^o -L kerl?. By 

Stone theorem iD is self-adjoint and in particular D is closed. Therefore by the 

closed range theorem ([10]), we have (kerZ)) = R (D) , where R (D) is the range 

of D. So there exists h S L,'J~,, such that Dh — </> — P^o- Moreover, since fyo—Vfyo 

I Jo | 

is even in v and the operator D reverses the parity in v, the function h is odd in v. 
Define /~ — f' h. We have 

ko = {Ao<fo,fa) = J |V0 O | 2 dx + An j j fo'ifo-Vfafdxdv 
= J |V0 o | 2 ^-8tt J J \Q(4>o-VM4>odxdv 



■ 47T 



\f' Q \{ct>o-V<f>ofdxdv 



^J-dxdv+2 J 0o J Df-dvdx + ^- J |V<A 

._i2 if if 

— — dxdv + — / (po^^dx + — / |V0o| 2 dx 
ol 27r 7 An J 



= 4tt y 1 </.rr/r ■■■ 2 / - V / DJ" ,lnlr-\- — / V. >, , \ ~ d.r 

= An (j J \2LLdxdv + (|V0 O | 2 - 2V0o ■ V0" ) 



where A</>~ = An J Df~dv. Notice that the last expression above is the Antonov 
functional AnH (f~, /")• Since /" is spherical symmetric and odd in v,we have 
H (f~ , f~) > by the proof in [55] which was further clarified in [33J and [2"Tj . 
Therefore we get ko > as desired and (Ao(j>, (f>) > ko \4>\^- 
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To get the estimate (|32|) . we rewrite 
(A cb, <f>)=e(J I V0| 2 dx + AnJ J f Q (</> - V<\>f dxdv^j + (1 - e) (4>& , 
> e 1 1 V0| 2 dx - 4tt£ ||0 - V<j>\? L 2 + (1 - e) fc 



Ml |^| 2 



>e / |V^di-87re|H|i a +(l-e)*o|^ (since ||P|| ia ^ < 1) 

1/61 " Wl 1/61 



> e y |V0| 2 + ((1 - a) k - Ce) \<j>\\ > e (J |V0| 2 dx + \<f>\ 
if e is small enough. 



□ 



Next, we will approximate the kerD by a finite dimensional approximation. Let 
{£i(E, L) = c*i(E)(3i(L)}°l 1 be a smooth orthogonal basis for the subspace ker D = 

{g(E,L)} C L,£,. Define the finite-dimensional projection operator Vn ■ L\'L\ ~ * 
|/o| \fo\ 

L \fl\ by 



JY 



(33) -PNh = J2( h ^i)\f^i 

i=i 

and the operator A : ii 2 — > L 2 by 

^ = -£^ + [4^ J f dv]cj>-A-K J f V N <t>dv- 
Lemma 3.2. There exists K , 5q > sucft £/ia£ when N > K we have 



(34) 

/or any € ii 2 . 

Proof. First we have j4 w 



Aq strongly in L 2 . In deed, for any (j> £ ii 2 , 



A N <t>-A M = 



4^/o {V N <t> - V<j>) dv 



<c\\v N <t>-v<t>\\ 







1/61 



as N — » cx^.We claim that for iV sufficiently large, the lowest eigenvalue of 
is at least fco/2 where ko > is the lowest eigenvalue of Aq. Suppose otherwise, 
then there exists a sequence {A„} and {4> n } C ii 2 with A„ < feo/2, ||0,i|| 2 
1 and A n (j) n — \ n (f> n . This implies that A</>„ is uniformly bounded in L 2 , by 
elliptic estimate we have ||<^n||/p < C for some constant C independent of n. 



Therefore there exists <po € ii 2 such that <p n 



weakly in if 2 . By the compact 
= 1. The 



5 o|| 2 



embedding of ii 2 L 2 , we have <f> n — > </>o strongly in L 2 and 
strong convergence of A™0o — > ^o^o implies that 

A™0„ -> A 4> Q 

weakly in I? . Let A n — > Ao < fco/2, then we have ^o^o = Ao</>0) a contradiction. 
Therefore we have (A N <f>, (f) > k /2 |0| 2 for S ii 2 , when N is large enough. The 
estimate (j34f is by the same proof of (|32|) in Lemma l3~T1 □ 
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Recalling © with f = [e E °- E -l]+ and Q (f) = (f + 1) ln(/ + l)-/, we further 
define functionals (related to the finite dimensional approximation of ker_D) as 

Mf)= { ai(-ln(s + l)+E )ds, 



Q i (f,L) = A i (f)p i (L), forl<i<N. 

for 1 < i < N. Clearly, 

diQi(fo, L) = ai {- ln(/ + 1) + Ea)Pi(L) = = £*(£, L), 

where {£,i(E , L)}f =1 are used to define Vn in Lemma 13.21 Define the Casimir 
functional (E < ) 

1(f) = f[Qo(f) + \\v\ 2 f-E f}dxdv-^ J \V<t>\ 2 dx 

which is invariant of the nonlinear Vlasov-Poisson system. We introduce additional 
N invariants 



Ji{f,L) = J Q l {f,L)dxdv. 
for 1 < i < N. We define £1 to be the support of fo(E). We first consider 

1(f) - I(fo) = J [Qo(f) - Qo(fo) + \\v\ 2 (f - fo) - E (f - f )]dxdv 
- / Vf/ • V((7 - C/ ) - i- / \V(U - U )\ 2 dx 
= J IQo(f) - Qo(/o) + (E- E )(f - f )]dxdv - i- J \V(U - U Q )\ 2 dx. 

Wc define 

9 = f-fo, <f>=U-U 

and 

(/ - /o)ln, ffout = (/ - /o)lfi<=, A0 in = J g in , A0 out = J g ou t ■ 

And we define the distance function for nonlinear stability as 
(35) 

d(f, fo) = y J [QoG?in + fo) - Qo(fo) + (E- E )g ia \dxdv\ + i- J \Vcb in \ 2 dx 



9h 



I I Qo(gout)dxdv + / (E - E Q )g out dxdv > 
J J Je>e J 

= din + — / \S7(j) in \ 2 dx + d out , 
8tt J 

for which each term is non- negative. We therefore split: 
1(f) ~ I(fo) 



<3o(/o + 9in) ~ Qo(fa) + (E- E )g- m ]dxdv 



^ J |V0 in | 2 dx| + 



J Qo(g ou t)dxdv + J (E- E )g out dxdv - ^- J \\7 (f> out \ 2 dx - ^- J V</> out • V<&, 
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In the estimates below, we use C,C',C" to denote general constants depending 
only on fo and quantities like ||/(i)|| iP (pG [l,+oo]) which equals ||/(0)|| iP and 
therefore always under control. We first estimate ||V0 ou t|| 2 to be of higher order of 
d, which also implies that J V0 out • V<f>i n dx is of higher order of d. 



Lemma 3.3. For e > sufficiently small, we have 

1 



|V</w| 2 dx < C (ed(f,fo) + p7a[d(/,/o)] 
Proof. In fact, since 

|V0 out | 2 dx < C\\ I g out dv\\ 2 L6/5 



5/3 



< C|| / ffout lE <E<E +£dv\\ L6/5 + C\\ I g out iE>E +edv\\ L6/5 



The first term is bounded by 

/[/.9o 2 ut^] 3/5 [/ 1 



E <E<E +s 



dv] 3/5 dx 



5/3 



< [ / 9 2 out dvdx] x 



[ 1 



E <E<E - 



,dv] 3/2 dx 



< Ce[ / 3out ducte] < Ce[ / g£ ut cfecfe 



2/3 



<Ced(f,f ). 

In the above estimates, we use that J J Qo(gout)dvdx > c J g 2 ut dvdx and 

J ^-E Q <E<E +edv < Ce, 

which can be checked by an explicit computation when e > is sufficiently small 
such that Eq + e < 0. 

On the other hand, by the standard estimates (see P21 P. 120-121]) 

|| J ffout 1_E>_E +E C M lie/5 
< f f ffout lE>E +edxdv 

(E - E )g out l E>Eo+£ dxdv 



\v\ 2 g ut lE>E +edxdv 



< 



< -d 



(E - E )g out l E>Eo + e dxdv + 2sup|J7 | / / g out l E>Eo+s dxdv 
2sup| C / | d y < C 



□ 
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By Lemma 13.31 we have 



< ||V0out|| 2 II 



in 1 1 2 



<c( £ 1 /3 d(/i/o) + _l_ [d(/)/o)] 4/3^ 

mall, 

C (e^d(f,f ) + -L.[d(fJ )}*/ 3 + -i^[d(/,/ )] 5 / 3 



and therefore for s sufficiently small, 
(36) I out > 

^out 

To estimate ii n , we split it into three parts 



?o(/o + 9m) - Qo(fo) + (E- E Q )g in + cf) in g in }dxdv 



8tt 



|V0 in | 2 dx 



(1 - r) / [Q (/o + 9in) - Qo(/o) + (-E - £ )5in + (I - P^inSin]^^ + s- / |V<^: 



8tt 



+ (1-t) I 



'N<j>m9mdxdv 



(37) 



= J 1 + J- 2 + Z- 3 

1 -Mil 1 ^im 

where A(/>; n — Air J g ln dv. We estimate each term in the following lemmas. 
Lemma 3.4. 

(38) 4 > T -d m ~Ct J \V(t> m \ 2 dx. 

Proof. In fact, since the integration region £1 is finite, we have 



II 



> T 



> T 



[Qo{fo + 9in) - Qo(fo) + (E - E )g in + 4> in g in ]dxdv + ^- [ |V</> in | 2 Gfo 

8tt J 

[Qo(/o + Sin) ~ QoCfo) + (E- E )g in ]dxdv - CtU in IU 6 ll3iii||,LS/5 
[Qo(/o +9in) - Qo(/o) + (E - E Q )g in ]dxdv - C"r||V0 in || i2 || 5in || 2 



> -rfi„-C"r||V0 in || 2 , 



since 



di„ = 



J [Qo(/o + Sin) - Qo(/ ) + - E ) ffin ]^ > C||<? in || 2 . 



□ 



To estimate If n , we need the following pointwise duality lemma from elementary 
calculus. 

Lemma 3.5. For any c, and any h, we have 

ffc/o CO = Qo(^ + /o) - <3o(/o) - Q' (/o)/i - eft > (/o + 1)(1 + c - e c ). 

Proof. Direct computation yields that the minimizer f c of g c j {h) satisfies the 
Euler-Lagrange equation 

ln(/ c + /o + l)-ln(/o + l)-c = 0, 
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SO 

U = {fo + 1) (e c - 1) ■ 
Thus by using the Euler-Lagrange equation, we deduce 

minffc/o (h) = 9c,d(fc) 

= (/c + /o + l)ln(l + / c + /o) 
- (/o + 1) ln(l + /„) - [1 + ln(/ + l)]/ c - c/ c 
= (fc + fo + l)[ln(l + fa + fo) - Hfo + 1) - c] 
+ / c In(l + /o) + c(/ + 1) - [1 + ln(/ + l)]/ c 
= (/o + l)(l + c-e c ). 



□ 



Lemma 3.6. 

(39) 



j2 > (l-r)S 



8tt 



\V(j) m \ z dx-Ce c ' d ^S 



Proof. Recall (f3"T|). By using Lemma 13.51 for c = — (<f)- m — Pat </>;„) and using the 
Taylor expansion, we have 

II = (l-T)J J [Q (/o + ffin) " Q (/o) + (-E - Bo)ffi n + (fax ~ P N <Pin) fin]dxd V 
+ ^( 1 - 7 ") / |V0in| 2 ^ 

- st (1 " r) / |v ^" |2da; + (1 - t) / / (/o + + ^ 

| y |V0 in | 2 cfe ~4irJ J |/g (£)| (0i„ ~ /Win) 2 dsdw 



-P: 



l )dxdv 



1-T 



|/^ (P)| |& n - Pat^I 3 dxcfe (Note (f (E) + 1) l n = \.fo(E)\) 



> 



(l-r)S 
8tt 



In the last line, we have used Lemma 13.21 To estimate the last term above and 
conclude our lemma, it suffices to show 

— PjV</>in|oo < CNd 2 . 

This follows from the facts that for the fixed N smooth functions £j, we have 



Ifi 



Win loo 



and since is spherically symmetric, 



l<M(r) 



u 2 Pin (u) du + up\ n (u) du 



<C'VR\ P in\ 2 <C"\\g in \\ 2 <C N d? 
where p nl — J g- m dv and R is the support radius of p m . 



□ 
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We now estimate the term J J PN<t>infindxdv, for which we use the additional 
invariants. 

Lemma 3.7. For any e > 0, we have 



(40) 



141 < C(d 1/2 (0) + e 1/2 rf 1/2 + -d)d 1/2 



Proof. By the definition of I? n in (|37p . it suffices to estimate (<7i n ,£i)- We expand 
Ji{f,L)-Ji(f ,L) 

= J»(/o + 9in, L) - Ji(fo, L) + Ji (gout, i) 

= (.9m ,&) +0(d) + Mg ut,L). 

Notice that 

\Ji(gout,L)\ <C\\g ovlt \\ L i < C||l{B <s<S +E}ffout|Ui +C r ||l{J5>B D +e}5out||i 1 

< e 1/2 ||5out||L 2 + j||l {E > So + E }( £ - £o)ffout|Ui < C[e 1/2 d 1/2 + 
It thus follows that 

|(Sin ,&)| < |^(/(Q),i) - Ji(/o,i)| +C[e 1 ^ 2 d 1 / 2 + -d] 

e 

Therefore 



PN<frmgin dxdv 



A 



A? 



i=l i=l 

<Cd 1 ^ 2 [d 1 / 2 (Q)+e 1 / 2 d 1 / 2 + ^d\. 
Now we prove the nonlinear stability of King model. 



□ 



Proof of Theorem The global existence of classical solutions of 3D Vlasov- 
Poisson system was shown in [M] for compactly supported initial data / (0) G C\ . 
Let the unique global solution be (/ (i) , <j> (t)). Let d (t) = d(f (t) , fo). Combining 
estimates ([38)l. (|39j) and flU, we have 



I(f(Q))-I(f )=I(f(t))-I(fo) 



> ^out + 2^ in 



(l-T)fr 

8tt 



Cr / |V0 in |^x 



C (e^d(t) + ^d(tf 3 + ^d(t) 5/3 ) - Ce^d^i 
Cd (t) 1/2 [^(O) + e 1 ' 2 d (t) 1/2 + -d (*)]. 
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Thus by choosing e and r sufficiently small, there exists 5' > such that 
(41) 

/(/ (0)) - I(f ) > 6'd(t) - C (d(t) 4/3 + d(t) 5/3 + d(i) 3/2 ) - Ce c ' d{t)h d{t)i 
-Cd {t) 1/2 d^ 2 (0). 

It is easy to show that /(/ (0)) — I(fo) < C"d(0). Define the functions j/i (x) = 
S'x 2 - Ce c " x x 3 - C (x 8 / 3 + x 10 / 3 + x 3 ) and y 2 (x) = Cd (0) 1/2 x + C"d (0). Then 

above estimates implies that yi ^d(i) 1 / 2 ^ < y 2 ^d(i) 1 ^ 2 ^. The function j/i is in- 
creasing in (0,xo) where xq is the first maximum point. So if d(0) is sufficiently 
small, the line y = y 2 (x) intersects the curve y = yi{x) at points x\,x 2 ,-" , 
with xi(d(0)) < x < x 2 (d(0)) < Thus the inequality y\{x) < y 2 (x) is 

valid in disjoint intervals [0, X\ (d (0))] and [x 2 (d (0)) , x 3 (d (0))], • • • . Because d (t) 

1 /2 

is continuous, we have that d(t) ' < x\ (d(0j) for all t < oo, provided we choose 
rf(O) 1 ' 2 < x . Since x\ (d(0)) — > as d(0) — > 0, we deduce the nonlinear stability 

1 /2 

in terms of the distance functional d(t) 1 . □ 

Acknowledgements 

This research is supported partly by NSF grants DMS-0603815 and DMS-0505460. 
We thank the referees for comments and corrections. 

References 

[1] Antonov, V. A. Remarks on the problem of stability in stellar dynamics. Soviet Astr, AJ., 4, 
859-867 (1961). 

[2] Antonov, V. A., Solution of the problem of stability of stellar system Emden's density law 
and the spherical distribution of velocities, Vcstnik Lcningradskogo Univcrsiteta, Leningrad 
University, 1962. 

[3] Arnold, V. I., Avcz, A., Ergodic problems of classical mechanics, W. A. Benjamin, Inc., New 
York-Amsterdam 1968. 

[4] Arnold, V. I., Mathematical methods of classical mechanics, Springer- Verlag, New York- 
Heidelberg, 1978. 

[5] Barnes, J.; Hut, P.; Goodman, J., Dynamical instabilities in spherical stellar systems, Astro- 
physical Journal, vol. 300, p. 112-131, 1986. 

[6] Bartholomew, P., On the theory of stability of galaxies, Monthly Notices of the Royal Astro- 
nomical Society, Vol. 151, p. 333 (1971). 

[7] Bertin, Giuseppe, Dynamics of Galaxies, Cambridge University Press, 2000. 

[8] Binney, J., Tremaine, S., Galactic Dynamics. Princeton University Press, 1987. 

[9] Doremus, J. P.; Baumann, G.; Feix, M. R., Stability of a Self Gravitating System with Phase 
Space Density Function of Energy and Angular Momentum, Astronomy and Astrophysics, 
Vol. 29, p. 401 (1973). 

[10] Gillon, D.; Cantus, M.; Doremus, J. P.; Baumann, G., Stability of self-gravitating spherical 
systems in which phase space density is a function of energy and angular momentum, for 
spherical perturbations, Astronomy and Astrophysics, vol. 50, no. 3, p. 467-470, 1976. 

[11] Fridman, A., Polyachenko, V., Physics of Gravitating System Vol I and II, Springer- Verlag, 
1984. 

[12] Glassey, Robert T., The Cauchy problem in kinetic theory, SIAM, Philadelphia, PA, 1996. 
[13] Goodman, Jeremy, An instability test for nonrotating galaxies, Astrophysical Journal, vol. 
329, p. 612-617, 1988. 

[14] Guo, Y., Variational method for stable polytropic galaxies, Arch. Rational Mech. Anal., 147, 
225-243, 1999. 



24 YAN GUO AND ZHIWU LIN 

[15] Guo, Y., On generalized Antonov stablility criterion for polytropic steady states, Contcm. 

Math., 263, 85-107, 1999. 
[16] Guo, Y., Rein, G., Stable steady states in stellar dynamics, Arch. Rational Mech. Anal., 147, 

no. 3, 225-243, (1999). 

[17] Guo, Y., Rein, G., Existence and stability of Camra type steady states in galactic dynamics, 

Indiana U. Math. J., 48, 1237-1255, 1999. 
[18] Guo, Y., Rein, G., Isotropic steady states in stellar dynamics, Commun. Math. Phys., 219, 

2001. 

[19] Guo, Y., Rein, G., Isotropic steady states in stellar dynamics revisited., Los Alamos Preprint, 
2002. 

[20] Hcnon, M., Numerical Experiments on the Stability of Spherical Stellar Systems, Astronomy 
and Astrophysics, Vol. 24, p. 229 (1973). 

[21] Guo, Y., Rein, G., Stability of the King Model and Symmetric Measure- Preserving Pertur- 
bations, Preprint. 

[22] Kandrup, H.; Signet, J. F.; A simple proof of dynamical stability for a class of spherical 
clusters. The Astrophys. J. 298, p. 27-33.(1985) 

[23] Kandrup, Henry E., A stability criterion for any collisionless stellar eguilibrium and some 
concrete applications thereof, Astrophysical Journal, vol. 370, p. 312-317, 1991. 

[24] King, Ivan R., The structure of star clusters. III. Some simple dynamical models, Astronom- 
ical Journal, Vol. 71, p. 64 (1966). 

[25] Lin, Zhiwu, Instability of periodicBG waves, Math. Res. Letts., 8, 521-534(2001). 

[26] Lin, Zhiwu and Strauss, Walter, Linear stability and instability of relativistic Vlasov- Maxwell 
systems, to appear in Comm. Pure Appl. Math. 

[27] Lin, Zhiwu and Strauss, Walter, Nonlinear stability and instability of relativistic Vlasov- 
Maxwell systems, to appear in Comm. Pure Appl. Math. 

[28] Lin, Zhiwu and Strauss, Walter, A sharp stability criterion for the Vlasov- Maxwell systems, 
submitted. 

[29] Lynden-Bell, D., The Hartree-Fock exchange operator and the stability of galaxies, Monthly 
Notices of the Royal Astronomical Society, Vol. 144, p. 189, 1969. 

[30] Lynden-Bell, D. Lectures on stellar dynamics. Galactic dynamics and N-body simulations 
(Thessaloniki, 1993), 3-31, Lecture Notes in Phys., 433, Springer, Berlin, 1994. 

[31] Merritt, David, Elliptical Galaxy Dynamics, The Publications of the Astronomical Society of 
the Pacific, Volume 111, Issue 756, pp. 129-168. 

[32] Palmer, P. L., Stability of collisionless stellar systems: mechanisms for the dynamical struc- 
ture of galaxies, Kluwer Academic Publishers, 1994. 

[33] Perez, Jerome and Aly, Jean-Jacques, Stability of spherical stellar systems - I. Analytical 
results, Monthly Notices of the Royal Astronomical Society, Volume 280, Issue 3, pp. 689- 
699, 1996. 

[34] Pfaffclmoser, K, Global classical solutions of the Vlasov- Poisson system in three dimensions 
for general initial data, J. Differential Equations 95 (1992), no. 2, 281-303. 

[35] Rein, G.: Collisionless Kinetic Equations from Astrophysics - The Vlasov- Poisson system. 
Preprint 2005. 

[36] Schaeffer, Jack, Steady states in galactic dynamics, Arch. Ration. Mcch. Anal. 172 (2004), 
no. 1, 1-19. 

[37] Sygnet, J. F.; des Forets, G.; Lachieze-Rey, M.; Pellat, R., Stability of gravitational systems 
and gravothermal catastrophe in astrophysics, Astrophysical Journal, vol. 276, p. 737-745, 
1984. 

[38] Wan, Y-H., On onlinear stability of isotropic models in stellar dynamics, Arch. Rational. 

Mech. Anal., 147, (1999) 245-268. 
[39] Wolansky, G., On nonlinear stability of polytropic galaxies. Ann. Inst. Henri Poincare. (1999), 

16, 15-48. 

[40] Yosida, Kosaku, Functional analysis, Sixth edition. Grundlehren der Mathcmatischcn Wis- 
senschaften, 123. Springer- Verlag, 1980. 



UNSTABLE AND STABLE GALAXY MODELS 



25 



Lefschetz Center for Dynamical Systems, Division of Applied Mathematics, Brown 
University, Providence, RI 02912, USA 
E-mail address: guoyScfm.brown.edu 

Mathematics Department, University of Missouri, Columbia, MO 65211 USA 
E-mail address: linamath.mlssourl.edu 



